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ABSTRACT 



Two alternative methods for optimizing an unconstrained non-linear 
function are investigated and compared. The investigations are made 
subject to a restriction as to the number of function evaluations 
available to conduct the optimization procedures. Powell's method of 
conjugate directions is employed as the direct search method and is 
considered the reference method. The alternate method is based on 
fitting a quadratic surface to the available function evaluations and 

i 

optimizing over the resulting fitted surface. The test functions 
considered in the investigation were limited to unimodal functions. 
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I. NATURE OF THE PROBLEM 



A. INTRODUCTION 

In Operations Research a major problem concerns the determination of 
the optimum response for an objective function. In many cases, the 
objective function may not lend itself to expression in closed analytical 
form. Information concerning the nature of the objective function may 
only be available through computer simulation, field testing or a combina- 
tion of both. Furthermore, restrictions may exist with respect to the 
quantity of information available or obtainable due to the costs involved 
in securing the data. Thus, it is incumbent that the most efficient use 
be made of the limited information available in order to achieve the best 
possible results in determining the optimum response. 

There exist several search techniques that could be adapted to solve 
the problem as stated above. Generally, these search techniques could 
be classified as direct search techniques or gradient search techniques. 
Direct search methods do not require the computation of partial deriva- 
tives of the objective function. Determination of the optimum response 
is based solely on values of the objective function itself. On the other 
hand, gradient search methods compute values of partial derivatives of 
the objective function. These resulting values are then used in select- 
ing future search directions. Search techniques employing gradient 
methods require more information (i.e., additional function evaluations, 
computer simulations, etc.) than direct search procedures. Thus, with 
the presence of restrictions on the quantity of information available it 
would be more advantageous to use a direct search method. 
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Direct Search methods can be broadly divided into three classifica- 
tions: tabulation methods, sequential methods and linear methods. 
Tabulation methods, as the name implies, result in a tabular listing of 
objective function responses for different values of the input variables. 
The "optimum" response is considered to be the "best" value found in the 
table. In the case of minimization problems, the "best" value would be 
the smallest value. An example of a tabulation method is random search. 

Sequential methods employ the use of geometric designs in the input 
variable space. The objective function is evaluated at the vertices of 
the geometric design. The sequence is repeated until the desired accuracy 
is achieved or the restriction of the quantity of information obtainable 
precludes additional testing. Factorial designs and the simplex method 
are examples of sequential methods. 

Linear methods employ the use of a set of direction vectors through- 
out the conduct of the search. The objective function is evaluated at 
different points along these search directions and conduct of the search 
is dependent upon the results obtained. The set of direction vectors 
may or may not be changed during the course of the search. Examples of 
linear methods are the alternating variable search and Powell's method 
of conjugate directions. 

An alternative approach to employing search techniques to solve the 
problem would be to use curve-fitting techniques. Using this technique, 
test points would be established in the input variable space and the 
objective function would be evaluated at these test points. A least- 
square polynomial surface would then be fitted to these resulting 
objective function values. To complete the procedure the location of 
the "optimum" response of the fitted surface would be determined and the 
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resulting input variable values would be input to the true objective 
function to determine the "optimum" response. 

B. STATEMENT OF THE PROBLEM 

The specific problem that was the subject of investigation was to 
compare the relative efficiency of a curve-fitting technique as opposed 
to a search technique. The assumption was made that a restriction exists 
on the quantity of information available (i.e., there v/ere a finite number 
of functional evaluation that could have been made). A direct result of 
the above assumption was that the investigation was limited to those 
techniques which do not require the computation of partial derivatives. 
Furthermore, the investigation was restricted to test objective functions 
of a unimodal nature insuring the existence of only one true optimum 
response. 
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II. EXPERIMENTAL PROCEDURES 



A. SEARCH PROCEDURE 

Powell's method of conjugate directions was chosen as the direct 
search technique to be used. Powell's method is considered to be one 
of the most efficient direct search techniques available [Ref.l]. Its 
efficiency is exceptionally good for functions that can be approximated 
by a quadratic in the region of the optimum. 

Powell's method commences with a search along a set of linearly 
independent directions that span the input variable space starting from 
an arbitrarily chosen initial point. The initial set of directions 
chosen are the co-ordinate directions. At the completion of each itera- 
tion a new direction is. defined and replaces one of the presently exist- 
ing directions only if the new set of directions is at least as efficient 
as the present set. If this is not the case, an additional iteration is 
conducted retaining the present set of vectors. A complete description 
of Powell's method and a detailed discussion of its efficiency and 
convergence is contained in [Ref. 2]. 

An existing version of Powell's method programmed in FORTRAN for use 
on the IBM 360 Computer was used as a subroutine in the overall computer 
program written to conduct the search technique optimization. The main 
program provided the calling argument required by the subroutine to 
conduct the optimization. 

B. CURVE-FITTING TECHNIQUE 

The curve-fitting technique used was based on standard multiple 
l-'near regression techniques [Refs. 3 and 4] modified as follows. 
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Because the test functions to be investigated were unimodal the decision 
was made to fit a quadratic surface to the experimental data. Thus, the 
order of the regression was double the number of input variables. 

An early decision required to be made concerned exactly what proced- 
ure would be used to place the test points in the input variable space. 

Two possibilities were considered. A completely random selection of the 
test points was one alternative. A second approach was to place the test 
points throughout the space in accordance with an experimental design. 

The first experimental design considered was a composite factorial design 
[Ref. 5]. This approach was attractive for objective functions of few 
input variables. However, the requirement for an excessive number of 
test points and consequently an increased number of function evaluations 
precluded the use of this design for objective functions of more than 
four variables. 

For functions of more than four variables it was decided to use an 
orthogonal design requiring 2n+l test points, where n is the number of 
input variables. The initial test point was placed in the center of the 
input variable space and the remaining points were placed at a fixed 
distance (0.5) in both the positive and negative direction along the 
co-ordinate axes passing through the center of the design. In the 
investigations a third alternative was also considered. For those cases 
where the number of observations (function evaluations) available exceeded 
the number required for the experimental design, the remaining points 
were located randomly in the input variable space. 

Once the test points were placed, the function was evaluated at these 
points. The value of input variables, the squares of the input variables 
and the value of the objective function for each test point were then 
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input to the regression subroutine to determine the regression co- 
efficients of the fitted surface. The resulting fitted surface was a 
quadratic of the form: 

n n 2 

f = E a- x- + £ b. + constant term 

i=l 1 1 i=l 1 1 

Provided the quadratic was positive (negative) definite an explicit 
determination of the minimum (maximum) value of the quadratic could be 
computed. In that case, the minimum (maximum) value of the quadratic 
occurs at the point: 

1 a i 

x.j — ~ ^ k ^ » i - 1 » . • • 

These values of x. were computed and substituted into the true objective 
function to determine the "optimal" objective function response.- 

Preliminary results indicated that the curve-fitting technique was 
generating a large number of indefinite quadratic surfaces. This situa- 
tion occurred more commonly when random choice was used to locate the 
test points. Any indefinite quadratics generated were not useful because 
it was not possible to find the location of the minimum response of the 
fitted surface. Thus, an "optimum" response could not be determined in 
those situations where the curve-fitting technique generated an indefinite 
quadratic. 

In an effort to overcome this problem, a procedure was introduced 
into the curve-fitting technique to reduce the number of observations 
considered by the regression procedure. The basic idea behind the pro- 
cedure was for the regression procedure to restrict its consideration 
to those test points that were close to the optimum. In that manner 
points at a distance from the optimum did not affect the fit. It was 
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felt that this procedure would insure a better fit in the area of the 
optimum and thereby, eliminate the number of indefinite quadratics 
generated. 

A search was made of the objective function values for each test 
point. The worst 25 percent of the test points were then discarded 
reducing the number of observations considered by the regression pro- 
cedure. Because the..curve-f itting procedure was designed to find the 
minimum objective response, the worst 25 percent of the test points were 
considered to be the 25 percent with the highest values of the objective 
function. This procedure was then repeated eliminating the worst 50 
percent of the test points. 

This entire procedure was programmed in FORTRAN for use on the IBM 

360 computer. The program as written is capable of handling up to 20 

' / 

input variables and 50 observations of test points. A detailed 
description of the curve-fitting program is contained in Appendix A. 

C. TEST FUNCTIONS 

Two types of objective functions were considered in the investigation 
quadratic functions, with cross product terms, and exponential functions. 
The decision to include cross product terms in the quadratic functions 
was made to preclude the possibility of achieving an exact fit when 
employing the curve-fitting technique. In the case of exponential func- 
tions, the exponent term was a quadratic function chosen so that the 
function itself was unimodal . 

The test functions used were positive definite quadratic functions. 
Thus, the optimum response was the minimum objective function value. 

The test functions were constructed so that this minimum occurred at 
values of the input variables restricted to the domain zero to one. 
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This restriction limited the selection of initial starting points for 
Powell's method to points lying within this domain. Test points for the 
curve-fitting technique were also restricted to the interval zero to one 
for all input variables. 

To insure that the quadratic functions of eight variables were in 
fact positive definite, the following procedure was used to generate 
eight by eight matrices of quadratic term coefficients. An eight by 
eight matrix was generated and pre-mul ti pi ied by its transpose. The 
resulting eight by eight matrix is assured to be positive definite. 

An explicit expression of all test functions considered can be found 
in Appendix B. 

D. NUMBER OF VARIABLES 

Functions of two different levels of input variables were considered 
in the investigation. Initially, functions of three input variables 
were investigated. The investigation concluded with a study of functions 
of eight input variables. While these two cases are not all inclusive, 
it was felt they gave a comparative indication of how well the two 
techniques handled functions of low and high levels of input variables. 

E. NUMBER OF OBSERVATIONS 

Again two different levels of observations were considered. The low 
level was fixed at 2n+l observations. This was the minimum number 
required for the orthogonal design used in the curve-fitting technique. 
Twice the number of observations in the low level was arbitrarily 
decided upon for the high level. An exception to this high level number 
was made in the case of functions of three input variables. An additional 
observation was added to the high level number to allow the use of a 
composite factorial design with the curve-fitting technique. 
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F. NUMBER OF TRIALS 

Ten trials were conducted for each procedure that was non-deter- 
ministic. These procedures included all Powell Searches and those curve- 
fitting techniques where random choice was used to locate some or all of 
the points. 

G. MEASURES OF EFFECTIVENESS 

Tv/o measures of effectiveness were employed to compare results: 

1. the mean optimum response attained by optimization technique, 

2. the mean Euclidean distance between the location of the true 
optimum and the location of the optimum response found by the 
optimization technique. 
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III. PRESENTATION OF RESULTS 



Tables I through XI contain the results of the eleven test functions 
investigated. The following definition of terms is provided to assist 
in understanding the tabulated data: 

TECHNIQUE - The name of the optimization technique used. 

POWELL - direct search technique, Powell's method of conjugate 
directions . 

C.F. DESIGN - curve-fit technique, employing experimental design 
to locate the test points. 

C.F. RANDOM - curve-fit technique, employing random choice to 
locate the test points. 

75 PERCENT C.F. DESIGN - curve-fit technique, employing experi- 
mental design to locate the test point, eliminating the 
worst 25 percent of the test points. 

50 PERCENT C.F. DESIGN - curve-fit technique, employing experi- 
mental design to locate the test points, eliminating 
the worst 50 percent of the test points. 

75 PERCENT C.F. RANDOM - curve-fit technique, employing random 
choice to locate the test points, eliminating the v/orst 
25 percent of the test points. 

50 PERCENT C.F. RANDOM - curve-fit technique, employing random 
choice to locate the test points, eliminating the v/orst 
50 percent of the test points. 

#0BS - The number of observations included in the investigation. 

OPTIMUM RESPONSE - The mean and standard deviation, where applicable, 

of the optimum response found using the optimization technique. 
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The mean and standard deviation were computed considering only 
the results associated with positive definite quadratics gene- 
rated by the curve-fitting technique. Results associated with 
indefinite quadratic fitted surfaces were meaningless. 

DISTANCE - The mean and standard deviation, where applicable, of the 
Euclidean distance between the location of the true optimum and 
the location of the optimum response using the optimization 
technique. The mean and standard deviation were computed con- 
sidering only the results associated with positive definite 
quadratics generated by the curve-fitting technique. Results 
associated with indefinite quadratic fitted surfaces were 
meaningless. 

REMARKS - This column contains the number of indefinite quadratic 
surfaces generated using the different curve-fitting techniques. 
In addition, curve-fitting techniques that yielded deterministic 
results are noted by the statement, 1 trial, in this column. 
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TABLE I 



3 VARIABLES 



QUADRATIC I 







TRUE 


OPTIMUM = 


-7.9875 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV 


REMARKS 


POWELL 


7 


-7.3566 


0.526 


0.381 


0.273 






15 


-7.5756 


0.277 


0.294 


0.218 




C.F. 


7 


-7.625 





0.170 





1 trial 


DESIGN 
















15 


-7.625 


-- 


0.170 


— 


1 trial 


C.F. 


7 


14.9451 


31.03 


1 .424 


1.369 


7 indef 


RANDOM 












quad 




15 


1 .6504 


24.01 


0.608 


1 .026 


2 i ndef 














quad 


75% C.F. 


15 


... 


— 


— 


_ _ 


1 trial 


DESIGN 












indef 














quad 


50% C.F. 


15 


-7.5283 


— 


0.116 





1 trial 


DESIGN 




• 










75% C.F. 


15 


-7.2643 


0.409 


0.297 


0.074 


7 indef 


RANDOM 












quad 


50% C.F. 


15 


-5.8791 


2.569 


0.551 


0.570 


7 indef 


RANDOM 












quad 


NOTES: 














1. C.F. 


design technique 


i achieved 


the best i 


result. Of 


interest is 



the fact that increasing the number of observations did not 
improve the results. 

2. C.F. random technique was quite erratic. An improvement accomp- 
anied an increase in the number of observations. However, when 
eliminating the worst 25% (50%) of the test points an improve- 
ment in mean optimum response and distance was accompanied by 
an increase in the number of indefinite quadratics. 
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TABLE II 



3 VARIABLES 



QUADRATIC 2 



TRUE OPTIMUM = -59.90 







OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


7 


-58.52 


1.02 


0.485 


0.167 






15 


-58.76 


0.816 


0.449 


0.169 




C.F. 

DESIGN 


7 


-58.14 


-- 


0.290 


— 


1 trial 




15 


-58.14 




0.290 




1 trial 


C.F. 


7 


-23.15 


35.73 


0.896 


0.365 


8 indef 


RANDOM 












quad 




15 


-59.03 


0.657 


0.274 


0.074 


6 indef 














quad 


75% C.F. 
DESIGN 


15 


7.77 


— 


0.919 


— 


1 trial 


50% C.F. 
DESIGN 


15 


210.60 


-- 


2.529 


— 


1 trial 


75% C.F. 


15 


-39.82 


_ _ 


0.885 


_ _ 


9 indef 


RANDOM 












quad 


50% C.F. 


15 


161 .83 


219.68 


1.619 


1 .235 


8 indef 


RANDOM 












quad 


NOTES : 














1 . Powel 1 1 s 


method achieved the best 


results 


at both 7 and 


15 



observations . 



2. At 15 observations the C.F. random techniques achieved better 
results than Powell's method. However, 6 out of 10 quadratics 
generated using this technique were indefinite. 

3. Eliminating the worst 25% (50%) of the test points did not re- 
flect any improvement using the C.F. design technique. Although 
the mean optimum response and distance improved when eliminating 
the worst 25% of the test points using the C.F. random technique, 
the improvement was accomplished at the expense of 3 more 
indefinite quadratics being generated. 
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TABLE III 



3 VARIABLES 
EXPONENTIAL 1 
TRUE OPTIMUM = 4.476 x 10 



OPTIMUM RESPONSE DISTANCE 



TECHNIQUE 


#OBS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


7 


6.599xl0" 3 


1 .71x1 O - 3 


0.478 


0.165 






15 


5.251xlO -3 


0.43x1 0~ 3 


0.336 


0.111 




C.F. 


7 


6.307xl0" 3 




0.366 




1 trial 


DESIGN 


15 


57 .487xlO~ 3 


-- 


0.766 




1 trial 


C.F. 


7 


5 . 491 xl 0 -3 


_ — 


0.337 


_ _ 


9 indef 


RANDOM 












quad 




15 


8.018x1 O -3 


2.70xl0" 3 


0.558 


. 0.268 


7 indef 














quad 


75% C.F. 


15 


_ _ 


— 


— 





1 trial 


DESIGN 












indef 

quad 


50% C.F. 
DESIGN 


15 


5 . 241 xl 0 _ 3 


-- 


0.389 




1 trial 


75% C.F. 


15 


1056. 026 


1829.075 


0.746 


0.563 


6 indef 


RANDOM 












quad 


50% C.F. 


15 


93.852xl0" 3 


1 45 . 24x1 0“ 3 


0.506 


0.142 


6 indef 


RANDOM 












quad 


NOTES: 














1. C.F. 


design technique 


achieved the 


best i 


results at 7 


observa- 



tions and the 50% C.F. design technique proved to be the best 
at 15 observations. 

2. At 7 observations C.F. random technique achieved a better mean 
optimum response than C.F. design technique. However, 9 out of 
10 quadratics generated were indefinite. 

3. Eliminating the worst 25% (50%) of the test points using the 
C.F. random technique decreased the number of indefinite quad- 
ratics generated. However, the mean optimum response in both 
cases became worse. 
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TABLE IV 



3 VARIABLES 



EXPONENTIAL 2 







TRUE 


OPTIMUM = 


0.0212 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


7 


0.0380 


0.0154 


0.606 


0.200 






15 


0.0285 


0.0088 


0.408 


0.249 




C.F. 

DESIGN 


7 


0.0306 


— 


0.374 


— 


1 trial 




15 


0.2548 


-- 


0.776 


— 


1 trial 


C.F. 


7 


0.0368 


0.0092 


0.694 


0.238 


6 indef 


RANDOM 












quad 




15 


0.0943 


0.0652 


0.641 


0.149 


6 indef 














quad 


75% C.F. 
DESIGN 


15 


7.0428 


— 


1 .128 


— 


1 trial 


50% C.F. 
DESIGN 


15 


0 . 0380 


— 


0.365 


— 


1 trial 


75% C.F. 


15 


1 .22xl0 7 


1 . 22x1 0 7 


1 .802 


1.026 


8 indef 


RANDOM 












quad 


50% C.F. 


15 


106.38 


148.94 


0.729 


0.302 


7 indef 



RANDOM quad 



NOTES: 

1. C.F. design technique was best at 7 observations. Powell's 
method achieved the best results at 15 observations. 

2. C.F. random techniques generated a large number of indefinite 
quadratics. Eliminating the worst 25% (50%) of the test points 
neither improved the mean optimum response or distance, nor 
did it decrease the number of indefinite quadratics. 
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TABLE V 



3 VARIABLES 



EXPONENTIAL 3 







TRUE 


OPTIMUM = 


0.2157 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


7 


0.5255 


0.313 


0.534 


0.205 






15 


0.2969 


0.056 


0.336 


0.110 




C.F. 


7 


0.2232 


_ M 


0.155 


_ _ 


1 trial 


DESIGN 
















15 


0.3423 




0.464 


-- 


1 trial 


C.F. 


7 


>10 76 


>10 76 


2.032 


2.252 


7 indef 


RANDOM 












quad 




15 


0.2873 


0.045 


0.357 


0.126 


6 indef 














quad 


75% C.F. 


15 


0.4668 


- - 


0.662 





1 trial 


DESIGN 














50% C.F. 


15 


0.2205 


_ _ 


0.164 


_ _ 


1 trial 


DESIGN 














75% C.F. 


15 


23.7196 


46.698 


0.625 


0.475 


5 indef 


RANDOM 












quad 


50% C.F. 


15 


0.3059 


0.073 


0.330 


0.054 


7 indef 


RANDOM 












quad 


NOTES: 














1. C.F. 


design technique 


: achieved 


the best l 


results at 7 


observa- 



tions. At 15 observations, C.F. design eliminating the worst 
50% of the test points achieved the best results. 

2. C.F. random techniques were extremely erratic generating many 
indefinite quadratics. In the case of 7 observations, one bad 
fit caused the mean optimum response to become so large as to 
be meaningless. 
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TABLE VI 









8 VARIABLES 














QUADRATIC 1 












TRUE 


OPTIMUM = - 


44.61 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


-41 .97 


1.207 


1.028 


0.047 






34 


-43.50 


0.591 


0.962 


0.047 




C.F. 


17 


-44.18 





0.644 


_ - 


1 trial 


DESIGN 
















34 


-27.26 


21.14 


1.871 


1.992 


2 indef 














quad 


C.F. 


17 


_ _ 


_ _ 


- - 


_ - 


10 indef 


RANDOM 












quad 




34 


-34.30 


7.76 


1.584 


1.844 


8 indef 














quad 


75% C.F. 


34 


-43.' 45 


1.569 


0.55 


0.066 


7 indef 


DESIGN 












quad 


50% C.F. 


34 


-43.14 


_ — 


0.827 





9 indef 


DESIGN 












quad 


75% C.F. 


34 


_ _ 


_ _ 


_ _ 


_ _ 


10 indef 


RANDOM 












quad 


50% C.F. 


34 


_ _ 


_ _ 






10 indef 


RANDOM 












quad 


NOTES: 














1. C.F. 


design 


achieved 


the best results at 


17 observations. At 



34 observations, Powell's method proved to be the best 
technique. However, it is significant to note that the 
results achieved using C.F. design techniques with 17 
observations is better than that achieved by Powell with 34 
observations . 

2. C.F. random techniques were a complete disaster generating 

all indefinite quadratic surfaces in 3 out of 4 investigations 
used . 
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TABLE VII 



8 VARIABLES 



QUADRATIC 2 







TRUE 


OPTIMUM = - 


91 .89 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


-79.56 


9.53 


0.997 


0.212 






34 


-85.04 


4.02 


0.942 


0.233 




C.F. 

DESIGN 


17 


15.31 


— 


0.763 


— 


1 trial 




34 


-16.92 


49.88 


1.099 


1 .314 


6 indef 














quad 


C.F. 


17 


_ _ 


« - 





_ _ 


10 indef 


RANDOM 












quad 




34 


676.47 





3.751 





9 indef 














quad 


75% C.F. 


34 


1358.07 


1308.57 


3.153 


1 .885 


8 indef 


DESIGN 












quad 


50% C.F. 


34 











_ — 


10 indef 


DESIGN 












quad 


75% C.F. 


34 


_ _ 






_ _ 


10 indef 


RANDOM 












quad 


50% C.F. 


34 








_ - 


_ _ 


10 indef 


RANDOM 












quad 


NOTES: 














1 . Powel 


1 ' s 


method achieved the best 


results 


at both 17 


and 34 



observations . 



2. All curve-fitting techniques were inefficient. A probable 
cause of this problem was the fact that the quadratic function 
investigated here had sharply rising contours in the vicinity 
of the optimum. 
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TABLE VIII 









8 VARIABLES 














QUADRATIC 3 












TRUE 


OPTIMUM = -68 


.25 










OPTIMUM 


RESPONSE 


DISTANCE 




TECHNIQUE 


ms 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


-43.29 


30.42 


1 .201 


0.454 






34 


-47.67 


26.62 


1 .135 


0.448 




C.F. 

DESIGN 


17 


154.19 


— 


1 .011 


— 


1 trial 




34 


1681.1 


3398.9 


14.68 


31.56 


3 indef 














quad 


C .F. 


17 





— 


-- 


— 


10 indef 


RANDOM 












quad 




34 





— 


* - 


— - 


10 indef 














. quad 


75% C.F. 


34 


1617.68 


1282.98 


2.893 


1 .495 


7 indef 


DESIGN 












quad 


50% C.F. 


34 


_ _ 





_ _ 


_ - 


10 indef 


DESIGN 












quad 


75% C.F. 


34 


_ _ 


_ _ 


_ _ 


— — 


10 indef 


RANDOM 












quad 


50% C.F. 


34 





_ - 







10 indef 


RANDOM 












quad 


NOTES: 














1 . Powel 1 1 s 


method achieved the best 


results 


at both 17 


and 34 



observations . 



2. All curve-fitting techniques were inefficient. A probable 

cause of this problem was the fact that the quadratic function 
investigated here had sharply rising contours in the vicinity 
of the optimum. 
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TABLE IX 







8 VARIABLES 












EXPONENTIAL 1 












TRUE OPTIMUM = 1 .3x1 0 -9 










OPTIMUM RESPONSE 


DISTANCE 




TECHNIQUE 


#0BS 


MEAN STD. DEV 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


1 245. 08x1 0 -9 3605. 4x1 0~ 9 


1 .072 


0.214 






34 


8.46x1 0~ 9 18. 38x1 0~ 9 


0.925 


0.188 




C.F. 


17 


2.3x1 0~ 9 — 


0.526 


_ _ 


1 trial 


DESIGN 














34 


10.6xl0~ y — 


0.674 




9 indef 












quad 


C.F. 


17 





— _ 


_ — 


10 indef 


RANDOM 










quad 




34 





_ „ 


_ _ 


10 indef 












quad 


75% C.F. 


34 


81 . 93x1 0~ 9 1 06.26x1 0 -9 


0.813 


0.038 


7 indef 


DESIGN 










quad 


50% C.F. 


34 


3.31x1 0 -9 


0.542 





9 indef 


DESIGN 










quad 


75% C.F. 


34 


__ 





_ _ 


10 indef 


RANDOM 










quad 


50% C.F. 


34 





_ _ 


_ _ 


10 indef 


RANDOM 










quad 


NOTES: 












1. C.F. 


desig 


n technique achieved the 


best 


results at 17 





observations while Powell's method proved best at 34 observa- 
tions. However, the results at 17 observations using C.F. 
design were better than those using Powell's method with 34 
observations . 

2. At 34 observations, C.F. design eliminating the worst 50% of 
the test points achieved results superior to Powell's method. 
However, 9 out of the 10 quadratics generated were indefinite. 

3. C.F. random techniques were completely inefficient generating 
indefinite quadratic surfaces in every situation they were 
used . 
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TABLE X 



8 VARIABLES 
EXPONENTIAL 2 
TRUE OPTIMUM = 1.02x10 



OPTIMUM RESPONSE DISTANCE 



TECHNIQUE 


ms 


MEAN 


STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


61 .27x1 0" 25 


1 08 . 1 3x1 0 -25 


1 .096 


0.190 






34 


14.40xl0" 25 


23 .61x1 0~ 25 


0.949 


0.204 




C.F. 

DESIGN 


17 

34 


33.4xl0" 25 


— 


0.591 


-- 


1 trial 

10 indef 
quad 


C.F. 

RANDOM 


17 


-- 


-- 


— 


-- 


10 indef 
quad 




34 


-- 


-- 


-- 


— 


10 indef 
quad 


75% C.F. 
DESIGN 


34 


1 8599x1 0" 25 


26296x1 0 -25 


1 .262 


0.495 


7 indef 
quad 


50% C.F. 
DESIGN 


34 


94. 3x1 0" 25 


-- 


0.696 


-- 


9 indef 
quad 


75% C.F. 
RANDOM 


34 


— 


-- 


-- 


-- 


10 indef 
quad 


50% C.F. 
RANDOM 


34 


-- 




-- 


-- 


10 indef 
quad 


NOTES: 















1. At 17 observations C.F. design achieved the best results while 
Powell's method was superior at 34 observations. 

2. C.F. random techniques were completely inefficient generating 
indefinite quadratic surfaces in every situation they were used. 
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TABLE XI 







8 VARIABLES 












EXPONENTIAL 3 












TRUE OPTIMUM = 9.' 


91 xl 0 -6 










OPTIMUM RESPONSE 


DISTANCE 




TECHNIQUE 


ms 


MEAN STD. DEV. 


MEAN 


STD. DEV. 


REMARKS 


POWELL 


17 


67.95xl0 -6 50.18xl0" 6 


1 .078 


0.269 






34 


1 7.0x1 0 -6 5.1 lxl 0" 6 


0.960 


0.277 




C.F. 


17 


1 2 . 92x1 0 -6 


0.516 


_ _ 


1 trial 


DESIGN 


34 


>10 26 >10 26 


1 .912 


1 .903 


4 indef 












quad 


C.F. 


17 





— 


— 


10 indef 


RANDOM 










quad 




34 1 


172.35xl0" 6 1147.92xl0" 6 


1.003 


0.349 


8 indef 












. quad 


75% C.F. 


34 


21 . 74x1 0 -6 1 2 . 09x1 0 -6 


0.608 


0.104 


2 indef 


DESIGN 










quad 


50% C.F. 


34 


1 3 . 63x1 0 -6 0.836xl0" 6 


0.540 


0.025 


6 indef 


DESIGN 










quad 


75% C.F. 


34 


10 6 


2.482 


_ _ 


9 indef 


RANDOM 










quad 


50% C.F. 


34 





_ _ 


_ _ 


10 indef 


RANDOM 










quad 


NOTES: 












1. C.F. 


design achieved the best results at 


17 observations. 



Although Powell's method proved to be the best at 34 observations 
the results achieved were not as good as those achieved using 
C.F. design with 17 observations. 

2. At 34 observations, C.F. design eliminating the worst 25% (50%) 
of the test points improved the mean optimum response and dist- 
ance. However, when eliminating the worst 25% a decrease in the 
number of indefinite quadratics occurred, while elimination of 
the worst 50% caused an increase in the number of indefinite 
quadratics . 

3. C.F. random techniques were inefficient due mainly to the 
large number of indefinite quadratics generated. 
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IV. CONCLUSIONS 



Overall, the direct search technique, Powell's method of conjugate 
directions, proved to be the best technique of those considered in the 
investigations. A particularly desirable feature of Powell's method was 
the fact that results achieved using this technique converged to the true 
optimum response when increasing the number of observations. Thus, 
Powell's method exhibited the characteristic that the more information 
(function evaluations) available, the better the results that could be 
expected to be achieved. A decrease in the distance between the loca- 
tion of the true optimum response and the location of the optimum found 
using Powell's method was also experienced as the number of observations 
was increased. 

The curve-fitting techniques, as a whole, were quite erratic. The 
biggest single problem area was the fact that the curve-fitting tech- 
niques generated a large number of indefinite quadratic surfaces. This 
situation was prevalent in both the case where the test points were 
located by random choice and the case where experimental design was 
used to locate the test points. However, it should be noted that the 
problem of indefinite quadratics arose in the curve-fitting techniques 
employing experimental design only in those cases where the total number 
of observations exceeded that number required by the experimental design. 
In this situation, the remaining number of test points were located by 
random choice. Thus, the results achieved strongly indicate a high 
correlation between the generation of indefinite quadratic surfaces 
and the use of random choice to locate any or all of the test points. 



27 



A second problem encountered when using curve-fitting techniques 
was the trend toward higher standard deviations of the optimum responses. 
This situation resulted from the fact that the curve-fitting techniques 
generated many bad fits with respect to the optimum response. This 
problem coupled with a reduction in the sample size due to the genera- 
tion of indefinite quadratics led to a higher variance of optimum 
response, thus a high standard deviation. 

To overcome these problems the effect of reducing the number of 
observations considered by the curve-fitting technique by elimination 
of the worst 25% and 50% of the test points was investigated. This 
approach did not yield consistent results. In some cases the number of 
indefinite quadratics generated was reduced, while in other cases the 
number remained the same or increased. In addition, in many cases where 
the number of indefinite quadratics was reduced, an increase in the 
number of bad fits with respect to the optimum response was experienced. 

However, curve-fitting' techniques do merit consideration under 
certain circumstances. Provided that the objective function did not 
have sharply rising contours in the immediate location of the true 
optimum, curve-fitting techniques employing experimental design, where 
the number of test points considered was exactly equal to the number 
required by the experimental design, provided results better than those 
achieved by Powell's method. In fact for some of the test functions 
investigated, the results achieved using the curve-fitting techniques 
employing experimental design were better than those achieved using 
Powell's method with twice the number of observations. 

Thus, on the whole Powell's method is an efficient method of 
optimization that consistently achieved better results than any of the 
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curve-fitting techniques investigated. Under certain circumstances, 
the curve-fitting technique employing experimental design to locate 
test points provided better results and was more efficient than Powell's 
method. 
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V. SUGGESTED FUTURE INVESTIGATIONS 



The problem of generating indefinite quadratic fits when using 
curve-fitting techniques was never overcome. In future investigations 
one way of addressing this problem would be to constrain the regression 
procedures so that the coefficients of the quadratic terms are all 
positive. Another method that could be used, either in connection with 
the above technique or by itself, is to force the fitted surface 
through a certain point. As a suggestion it might prove useful to 
force the fitted surface through the best of the test points. The best 
test point being chosen as the one with the smallest function value in 
the case of a minimization problem. 

Another interesting question that could be considered is whether 
a combination of the search and curve-fitting procedures would achieve 
better results than either procedure achieved by itself. A suggested 
approach to this question would be to use Powell's search initially to 
find the general area of the optimum solution. Once this has been 
achieved, the curve-fit design procedure could be used in this general 
area to obtain the optimum solution. 
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APPENDIX A 



CURVE-FITTING PROGRAM 



The purpose of the curve-fitting program was threefold: 

1) determine the location of the test points and their respective 
functional values, 

2) fit a quadratic surface to these test points, 

3) determine the location of the optimum response of the resulting 
"fitted" quadratic surface and compute the true functional 
value at that point. 

A. LOCATION OF TEST POINTS 

The program considered two different methods for locating test 
points in the input variable space, random choice or placement in 
accordance with an experimental design. Input data to the program pro- 
vided the basis for selection of the appropriate method. A uniform 
random number generator was used in connection with the method of 
random choice. The two experimental designs used, composite factorial 
and orthogonal, were written into the program to be constructed by the 
computer as they were needed. 

Upon selection of the appropriate method a matrix was constructed, 
each row of the matrix containing one observation of the input variables. 
At this stage the number of columns in the matrix was equal to the number 
of input variables. The size of the matrix was then expanded in the 
following manner. For each observation the value of each input variable 
was squared and stored in the corresponding row of the matrix, thus, 
doubling the number of columns of the matrix. A final column was added 
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to the matrix by computing the function value corresponding to each 
observation of the input variables. 

The completed matrix was then input to the curve-fitting procedure 
unless a reduction in the number of observations was called for. In 
that case the function values were scanned by the computer to determine 
the worst 25% or 50% of the values as appropriate. These worst function 
values along with their corresponding input variables and their squares 
were eliminated from the matrix reducing the number of rows in the 
matrix input to the curve-fitting procedure. 

B. FITTING THE QUADRATIC SURFACE 

Multiple linear regression techniques were used to fit a quadratic 
surface to the test points. This section of the program consisted of 
three subroutines and the procedures for computation of the regression 
coefficients. The three subroutines used were CORRE, ORDER, and MINV. 
These subroutines are included in the IBM 360 Scientific Subroutine 
Package and are stored internally on the computer. 

The purpose of subroutine CORRE was to compute the means, the 
standard deviations and a matrix of correlation coefficients of the 
variables included in the regression. In this case, the original input 
variables, their squares and the function values. The matrix constructed 
in the initial section of the program served as input to CORRE to accomp- 
lish these tasks. 

The resulting correlation coefficient matrix from subroutine CORRE 
was then input to subroutine ORDER. The function of subroutine ORDER 
was to compute a matrix containing the correlation coefficients of the 
independent variables and a vector containing the correlation coefficients 
of the independent variables with the dependent variables. In the 
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program, the original input variables and their squares were designated 
the independent variables, and the function value was designated the 
dependent variable. 

To check the validity of the regression the matrix of correlation 
coefficients among independent variables calculated by subroutine ORDER 
was input to subroutine MINV. Subroutine MINV computed the determinant 
of this matrix. If the determinant was non-zero, the regression was 
valid and the program continued. However, if the determinant was zero 
the program terminated due to the presence of multi-col 1 inearity in the 
regression. 

Provided the regression was valid, the next step was to compute the 
regression coefficients of the independent variables. To accomplish 
this task a portion of subroutine MULTR, also included in the IBM 360 
Scientific Subroutine Package, was used. The standard deviations com- 
puted by subroutine CORRE, and the matrix of correlation coefficients 
among the independent variables and the vector of correlation 
coefficients between the independent variables and the dependent 
variable computed by subroutine ORDER served as input to this subroutine. 
The output of the subroutine was a vector containing the regression 
coefficients. 

C. LOCATION OF OPTIMUM RESPONSE 

The vector of regression coefficients contained the necessary 
information to determine the location of the optimum response of the 
fitted surface. The length of this vector was 2n, where n is the number 
of input variables. The resulting fitted quadratic was of the form: 

n n p 

* f = z a. x. + z b. xf + constant 

i=l 1 1 i=l 1 1 
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The value of the a-'s was contained in the first n elements of the vector 

I / 

of regression coefficients and the value of the b_j ' s was contained in 
the last n elements. 

The location of the optimum (minimum) response was determined as 
follows. The values: 

v = 1 !i . , = , 

X • k 3* I J • • • > *1 J 

i 

were computed. These values uniquely determined the location of the 
minimum response of the fitted quadratic provided that the quadratic 
was positive definite. A vector containing the location of the optimum 
response of the fitted surface was then substituted into the true 
objective function to determine the optimum objective response for the 
curve-fitting procedures. 
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APPENDIX B 



TEST FUNCTIONS 



I. THREE VARIABLE FUNCTIONS 



A. QUADRATIC I 



2 1 2 


X + X T 


-6 


4 3 




-8 


6 




-13 



True Optimum = -7.9875 At X-j = 0.625 

X 2 = 0.4 
X 3 = 0.725 

B. QUADRATIC 2 



1 — 
X 

II 

u_ 


29 26 


30 


X + X T 


-80.2 




29 


34 




-81 .0 






45 




-98.4 



True Optimum = -59.90 



C. EXPONENTIAL 1 



At X 1 = 0.7 
X 2 = 0.3 
X 3 = 0.4 



F = exp 




2 3 
2 2 
4 



X + X T 




True Optimum = 0.004476 



D. EXPONENTIAL 2 



At X 1 = 0.2727 
X 2 = 0.8636 
X 3 = 0.3636 



F = exp / X T 


8 2 
2 


-3 

-2 


\ 




4 




True Optimum = 0.0212 



At X ] = 0.6364 
X 2 = 0.9318 
X 3 = 0.8182 
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E. EXPONENTIAL 3 



F - exp j X T 
True Optimum 



A. QUADRATIC 1 

F = X T [4 2 

2 



True Optimum 



B. QUADRATIC 2 

F = X T [93 -53 

103 



True Optimum = 



8 2-3 

2 -2 
4 



X + X T 




= 0.2157 



At X 1 = 0.2273 
X 2 = 0.8864 
X 3 = 0.3636 



II. EIGHT VARIABLE FUNCTIONS 



4 2 2 

3 3 0 

9 6 5 

7 0 

11 



= -44.61 



4 2 0 


X + X T 


-18.8 


1 2 0 




-13.6 


5 4 2 




-38.0 


2 1 1 




-24.4 


6 6 0 




-25.8 


9 0 0 




-23.8 


12 0 




-24.8 


5 




j_- 1 0 . 4 



At X 1 = 0.6 X 5 = 0.7 

X 2 = 0.9 X 6 = 0.2 

X 3 = 0.2 X ? = 0.3 

X 4 = 0.8 Xg = 0.8 



54 32 13 31 -67 43 


X + X T 


-96.2 


54 -40 -28 1 46 -1 




-75.0 


50 -28 -1 -20 54 -30 




6.4 


22 8 7 -30 14 




3.2 


32 -11 3 -15 




4.0 


25 -30 30 




-65.8 


69 -45 




48.0 


57 




-89.4 : 



-91 .89 



At X 1 = 0.7 X 5 = 0.6 

X 2 = 0.7 X g = 0.8 

X 3 = 0.4 X ? = 0.2 

X 4 = 0.3 Xg = 0.3 
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QUADRATIC 3 
F = X T 



43 -23 16 -29 25 -22 22 -21 


X + X T 


- 1 .0 


50 -22 53 -45 39 -39 37 




- 78.0 


19 -28 24 -21 21 -20 




32.8 


106 -69 60 -60 57 




-146.2 


76 -45 45 -43 




92.0 


41 -40 37 




- 85.2 


45 -35' 




79.6 


38 




- 83.0 



True Optimum = -68.25 



At X 1 = 

x 2 - 

Xo = 



0.7 

0.1 

0.2 

0.4 



X 5 

X 6 

X 7 

X 0 = 



0.1 

0.6 

0.2 

0.6 



D. EXPONENTIAL 1 



F = exp 



V 


5 1 -3 0 2 0 0 0 


X + X T 


-11.4 






2 -2 0 3 0 0 0 




- 7.6 






4 0-2000 




8.0 






10 0 4 -4 1 




-16.4 






6 0 0 0 




-15.0 






3 -2 1 




- 7.4 






8 -3 




1 .2 






4 




- 2.0 


J 



True Optimum = 1.3 x 10 



-9 



At X 1 = 0.9 
X 2 = 0.7 
X 3 = 0.3 
X 4 = 0,9 



X 5 = 0.7 

X 6 = °’ 3 
X y = 0.6 

X 8 = °- 4 



EXPONENTIAL 2 



F = exp 



T 




T 






1/144 X 1 


806 374 473 390 456 388 570 620 


X + X 1 


-25.84 






395 453 417 585 275 405 452 




-20.95 






587 529 696 417 583 535 




-27.40 






810 574 297 403 589 




-25.00 






1581 411 644 624 




-34.75 






385 591 372 




-21.61 






1574 693 




-42.48 




V 


930 




-32.72 J 


IJ 
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True Optimum = 1.02 x 10 



At X 1 = 0.1032 
X 2 = 0.1502 
X 3 = 0.1759 
X 4 = 0.3889 



X 5 = 0.3980 
X 6 = 0.7504 
X 7 = 0.9039 
Xg = 0.8039 



F. EXPONENTIAL 3 



( J 

1/144 X 1 


295 171 -170 123 125 128 5- 24 


X + X T 


-4.48 


'N 




687 - 90 136 -138 89 138-191 




-6.76 






316 -140 -150 28 196- 44 




-1 .63 






302 - 42 149 -192 14 




-2.27 






329 199 -108 114 




-1 .28 






687 -125 122 




-10.40 






707 42 




-7.53 






269 




-1 .84 




V 








) 



True Optimum = 9.91 x 10 


At X ] = 0.4004 
X 2 = 0.4158 


X 5 = 0.4314 
X 6 = 0.7791 




X 3 = 0.6355 


X 7 = 0.8928 




X 4 = 0.7187 


Xg = 0.2144 
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